<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
  <meta http-equiv="Content-Style-Type" content="text/css" />
  <meta name="generator" content="pandoc" />
  <title>linux内核debug调试</title>
  <style type="text/css">code{white-space: pre;}</style>
  <style type="text/css">
div.sourceCode { overflow-x: auto; }
table.sourceCode, tr.sourceCode, td.lineNumbers, td.sourceCode {
  margin: 0; padding: 0; vertical-align: baseline; border: none; }
table.sourceCode { width: 100%; line-height: 100%; }
td.lineNumbers { text-align: right; padding-right: 4px; padding-left: 4px; color: #aaaaaa; border-right: 1px solid #aaaaaa; }
td.sourceCode { padding-left: 5px; }
code > span.kw { color: #007020; font-weight: bold; } /* Keyword */
code > span.dt { color: #902000; } /* DataType */
code > span.dv { color: #40a070; } /* DecVal */
code > span.bn { color: #40a070; } /* BaseN */
code > span.fl { color: #40a070; } /* Float */
code > span.ch { color: #4070a0; } /* Char */
code > span.st { color: #4070a0; } /* String */
code > span.co { color: #60a0b0; font-style: italic; } /* Comment */
code > span.ot { color: #007020; } /* Other */
code > span.al { color: #ff0000; font-weight: bold; } /* Alert */
code > span.fu { color: #06287e; } /* Function */
code > span.er { color: #ff0000; font-weight: bold; } /* Error */
code > span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */
code > span.cn { color: #880000; } /* Constant */
code > span.sc { color: #4070a0; } /* SpecialChar */
code > span.vs { color: #4070a0; } /* VerbatimString */
code > span.ss { color: #bb6688; } /* SpecialString */
code > span.im { } /* Import */
code > span.va { color: #19177c; } /* Variable */
code > span.cf { color: #007020; font-weight: bold; } /* ControlFlow */
code > span.op { color: #666666; } /* Operator */
code > span.bu { } /* BuiltIn */
code > span.ex { } /* Extension */
code > span.pp { color: #bc7a00; } /* Preprocessor */
code > span.at { color: #7d9029; } /* Attribute */
code > span.do { color: #ba2121; font-style: italic; } /* Documentation */
code > span.an { color: #60a0b0; font-weight: bold; font-style: italic; } /* Annotation */
code > span.cv { color: #60a0b0; font-weight: bold; font-style: italic; } /* CommentVar */
code > span.in { color: #60a0b0; font-weight: bold; font-style: italic; } /* Information */
  </style>
  <link rel="stylesheet" href="doc/linux-style.css" type="text/css" />
</head>
<body>
<div id="header">
<h1 class="title">linux内核debug调试</h1>
</div>
<div id="TOC">
<ul>
<li><a href="#linux内核debug调试">linux内核debug调试</a><ul>
<li><a href="#一bpf">一、BPF</a></li>
<li><a href="#二在ubuntu-22.04上安装bcc的完整指南">二、在Ubuntu 22.04上安装BCC的完整指南</a></li>
<li><a href="#使用bpftrace-打印call_timeout的调用栈">使用bpftrace 打印call_timeout的调用栈</a></li>
</ul></li>
</ul>
</div>
<h1 id="linux内核debug调试">linux内核debug调试</h1>
<h2 id="一bpf">一、BPF</h2>
<ul>
<li><ol style="list-style-type: decimal">
<li>BPF全称是「Berkeley Packet Filter」,翻译过来是「伯克利包过滤器」。</li>
</ol></li>
<li><ol start="2" style="list-style-type: decimal">
<li>那么为什么需要BPF呢？这就得提到很多年前的程序都是作为用户级进程进行的，为了分析只在内核空间的数据，这些数据都得从内核空间拷贝到用户空间的内存中，因此会有大量的开销。</li>
</ol></li>
<li><ol start="3" style="list-style-type: decimal">
<li>随着现在的数据增多，不得不放弃在用户空间分析内核数据，因而BPF应运而生——一种在内核空间执行高效安全的程序的机制。这里BPF的原理就不深究了，具体看怎么使用。</li>
</ol></li>
</ul>
<pre><code>先安装bpftrace：
    apt-get install bpftrace  #ubuntu系统
    dnf install bpftrace   ##fedroa系统
我通过ubuntu的apt安装的是0.14发行版，但bpftrace会出现&quot;段错误&quot;。我源码安装都没用，太🍀蛋了！！！
网上查阅到的解析就是，你再安装一遍新版本，v-0.14版本太老了。下载APPImage安装吧。</code></pre>
<p><a href="https://github.com/bpftrace/bpftrace/discussions/3026">bpf -0.14 段错误官方回答</a></p>
<div class="sourceCode"><pre class="sourceCode sh"><code class="sourceCode bash">编写<span class="ex">test.bt</span>:
<span class="ex">kprobe</span>:ext2_read_folio
<span class="kw">{</span>
        <span class="ex">@start</span>[tid] = nsecs<span class="kw">;</span>
        <span class="bu">printf</span>(<span class="st">&quot;kprobe\n&quot;</span>);
        <span class="ex">print</span>(kstack());
<span class="kw">}</span>

<span class="ex">kretprobe</span>:ext2_read_folio
<span class="kw">{</span>
        <span class="va">$us</span> = <span class="kw">(</span><span class="ex">nsecs</span> - @start[tid]<span class="kw">)</span> <span class="ex">/</span> 100<span class="kw">;</span>
        <span class="bu">printf</span>(<span class="st">&quot;kretprobe, duration %d\n&quot;</span>, <span class="va">$us</span>);
        <span class="ex">delete</span>(@start[tid]);
        <span class="ex">print</span>(kstack());
<span class="kw">}</span>
然后执行命令
<span class="ex">bpftrace</span> test.bt <span class="kw">&amp;</span>
<span class="ex">mkfs.ext2</span> -F image
<span class="fu">mount</span> image /mnt
<span class="bu">echo</span> something <span class="op">&gt;</span> /mnt/file
<span class="bu">echo</span> 3 <span class="op">&gt;</span> /proc/sys/vm/drop_caches
<span class="fu">cat</span> /mnt/file</code></pre></div>
<p>test.bt 脚本通过以下方式跟踪 ext2_read_folio 脚本中包含两个探针（kprobe 和 kretprobe），记录进入read函数和出read函数的时间戳，并计算。</p>
<h2 id="二在ubuntu-22.04上安装bcc的完整指南">二、在Ubuntu 22.04上安装BCC的完整指南</h2>
<p>BCC (BPF Compiler Collection) 是一个用于创建高效内核追踪和操作工具的工具包，基于扩展的伯克利包过滤器(eBPF)技术。本指南详细介绍了在Ubuntu 22.04上安装和验证BCC的步骤。</p>
<h3 id="方法一使用ppa源安装推荐">方法一：使用PPA源安装（推荐）</h3>
<p>这是最简单快捷的安装方法，适用于大多数用户。</p>
<h4 id="步骤1添加bcc-ppa源">步骤1：添加BCC PPA源</h4>
<div class="sourceCode"><pre class="sourceCode sh"><code class="sourceCode bash"><span class="fu">sudo</span> apt-add-repository ppa:hadret/bpfcc
<span class="fu">sudo</span> apt-get update</code></pre></div>
<h4 id="步骤2安装bcc包">步骤2：安装BCC包</h4>
<div class="sourceCode"><pre class="sourceCode sh"><code class="sourceCode bash"><span class="fu">sudo</span> apt-get install -y bpfcc-tools python3-bpfcc</code></pre></div>
<h4 id="步骤3验证安装">步骤3：验证安装</h4>
<div class="sourceCode"><pre class="sourceCode bash"><code class="sourceCode bash"><span class="co"># 列出已安装的BCC工具</span>
<span class="fu">find</span> /usr -name <span class="st">&quot;*-bpfcc&quot;</span> -type f <span class="kw">|</span> <span class="fu">sort</span>

<span class="co"># 验证Python模块</span>
<span class="ex">python3</span> -c <span class="st">&quot;import bcc; print(bcc.__file__)&quot;</span>

<span class="co"># 测试运行BCC程序（例如execsnoop）</span>
<span class="fu">sudo</span> execsnoop-bpfcc -h</code></pre></div>
<h3 id="方法二从源代码编译安装">方法二：从源代码编译安装</h3>
<p>如果你需要最新版本或自定义构建，可以从源代码安装。</p>
<h4 id="步骤1安装依赖项">步骤1：安装依赖项</h4>
<div class="sourceCode"><pre class="sourceCode bash"><code class="sourceCode bash"><span class="co"># 安装基本依赖</span>
<span class="fu">sudo</span> apt-get update
<span class="fu">sudo</span> apt-get install -y bison build-essential cmake flex git libedit-dev \
  libllvm14 llvm-14-dev libclang-14-dev python3 zlib1g-dev libelf-dev \
  libfl-dev python3-pip

<span class="co"># 安装其他依赖</span>
<span class="fu">sudo</span> apt-get install -y arping clang-14 dh-python libdbus-1-dev libluajit-5.1-dev \
  luajit libpcap-dev libpython3-dev libzip-dev python3-netaddr python3-pyroute2

<span class="co"># 安装内核头文件</span>
<span class="fu">sudo</span> apt-get install -y linux-headers-<span class="va">$(</span><span class="fu">uname</span> -r<span class="va">)</span></code></pre></div>
<h4 id="步骤2克隆bcc仓库">步骤2：克隆BCC仓库</h4>
<div class="sourceCode"><pre class="sourceCode sh"><code class="sourceCode bash"><span class="bu">cd</span> ~
<span class="fu">git</span> clone https://github.com/iovisor/bcc.git
<span class="co"># 如果遇到网络问题，可以使用国内镜像：</span>
<span class="co"># git clone https://gitee.com/mirrors/bcc.git</span></code></pre></div>
<h4 id="步骤3编译和安装">步骤3：编译和安装</h4>
<div class="sourceCode"><pre class="sourceCode sh"><code class="sourceCode bash"><span class="fu">mkdir</span> -p ~/bcc/build <span class="kw">&amp;&amp;</span> <span class="bu">cd</span> ~/bcc/build
<span class="fu">cmake</span> -DCMAKE_INSTALL_PREFIX=/usr -DPYTHON_CMD=python3 ..
<span class="fu">make</span> -j<span class="va">$(</span><span class="ex">nproc</span><span class="va">)</span>
<span class="fu">sudo</span> make install</code></pre></div>
<h4 id="步骤4验证安装">步骤4：验证安装</h4>
<div class="sourceCode"><pre class="sourceCode sh"><code class="sourceCode bash"><span class="co"># 验证Python模块</span>
<span class="ex">python3</span> -c <span class="st">&quot;import bcc; print(bcc.__file__)&quot;</span>

<span class="co"># 测试BCC工具</span>
<span class="fu">sudo</span> /usr/share/bcc/tools/execsnoop</code></pre></div>
<h3 id="创建并运行自己的bcc程序">创建并运行自己的BCC程序</h3>
<div class="sourceCode"><pre class="sourceCode python"><code class="sourceCode python"><span class="co">#!/usr/bin/python3</span>
<span class="co"># 文件名: test_bcc.py</span>
<span class="im">from</span> bcc <span class="im">import</span> BPF

<span class="co"># 定义BPF程序</span>
program <span class="op">=</span> <span class="st">&quot;&quot;&quot;</span>
<span class="st">int hello(void *ctx) {</span>
<span class="st">    bpf_trace_printk(&quot;Hello, BCC!</span><span class="ch">\\</span><span class="st">n&quot;);</span>
<span class="st">    return 0;</span>
<span class="st">}</span>
<span class="st">&quot;&quot;&quot;</span>

<span class="co"># 加载BPF程序</span>
b <span class="op">=</span> BPF(text<span class="op">=</span>program)
b.attach_kprobe(event<span class="op">=</span>b.get_syscall_fnname(<span class="st">&quot;clone&quot;</span>), fn_name<span class="op">=</span><span class="st">&quot;hello&quot;</span>)

<span class="co"># 打印头部</span>
<span class="bu">print</span>(<span class="st">&quot;正在跟踪clone()系统调用...&quot;</span>)
<span class="bu">print</span>(<span class="st">&quot;按Ctrl+C退出&quot;</span>)

<span class="co"># 读取并打印trace输出</span>
<span class="cf">try</span>:
    b.trace_print()
<span class="cf">except</span> <span class="pp">KeyboardInterrupt</span>:
    <span class="cf">pass</span></code></pre></div>
<p>运行此脚本：</p>
<div class="sourceCode"><pre class="sourceCode sh"><code class="sourceCode bash"><span class="fu">chmod</span> +x test_bcc.py
<span class="fu">sudo</span> ./test_bcc.py</code></pre></div>
<h3 id="常用bcc工具">常用BCC工具</h3>
<p>BCC安装后提供了许多强大的工具，包括：</p>
<ul>
<li><code>execsnoop-bpfcc</code> - 跟踪新进程的执行</li>
<li><code>opensnoop-bpfcc</code> - 跟踪open()系统调用</li>
<li><code>tcptracer-bpfcc</code> - 跟踪TCP连接</li>
<li><code>tcpconnect-bpfcc</code> - 跟踪主动TCP连接</li>
<li><code>tcpaccept-bpfcc</code> - 跟踪被动TCP连接</li>
<li><code>biosnoop-bpfcc</code> - 跟踪块I/O延迟</li>
<li><code>funccount-bpfcc</code> - 统计函数调用次数</li>
<li><code>stackcount-bpfcc</code> - 统计内核堆栈轨迹</li>
</ul>
<p>使用这些工具时需要加上<code>sudo</code>权限。</p>
<h3 id="参考资料">参考资料</h3>
<ul>
<li><a href="https://github.com/iovisor/bcc">BCC官方GitHub仓库</a></li>
<li><a href="https://github.com/iovisor/bcc/blob/master/docs/reference_guide.md">BCC参考指南</a></li>
<li><a href="https://www.brendangregg.com/ebpf.html">Linux eBPF追踪工具</a></li>
</ul>
<h2 id="使用bpftrace-打印call_timeout的调用栈">使用bpftrace 打印call_timeout的调用栈</h2>
<p><code>sudo bpftrace -e 'kprobe:call_timeout { printf(&quot;Caller: %s\n&quot;, kstack); }'</code></p>
<p>sudo bpftrace -e 'kprobe:xprt_set_retrans_timeout_rtt { printf(&quot;Caller: %s&quot;, kstack); }'</p>
</body>
</html>
